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available  from  a  website  (http://ctat.pact.cs.cmu.edu/).  We  estimate  that  there  are  currently 
between  100  and  200  users  of  CTAT. 

Accomplishments: 

We  were  able  to  show  that  our  tools  can  be  used  to  build  many  different  rule-based  cognitive 
models,  such  as  a  Warrior  Simulator  at  Fort  Benning  (Livak  &  Heffeman,  2004),  a  Logic  Tutor 
for  internal  use  at  CMU,  and  a  Genetics  tutor. 

In  preliminary  controlled  experiments  involving  basic  Cognitive  Tutor  development  tasks, 
we  found  efficiency  gains  due  to  CTAT  of  1 .4  to  2  times  faster  (Aleven,  McLaren,  Sewall  & 
Koedinger,  2006).  We  also  demonstrated  across  4  different  behavioral  modeling  projects  that 
our  tools  created  example-tracing  tutors  that  drastically  reduced  modeling  costs  (Koedinger, 
Aleven,  Heffeman,  McLaren,  &  Hockenberry,  2004,  Heffeman,  Turner,  Lourenco,  Macasek, 
Nuzzo-Jones,  Koedinger,  2006).  Not  only  did  we  reduce  the  time  dramatically  (averaging  a 
reduction  of  over  a  factor  5),  we  also  reduced  the  experience  level  modelers  needed. 

Finally,  we  pushed  the  state  of  the  art  in  data  mining  and  machine  learning  support  to  help 
rule-writers  (McLaren  et  al  2005;  Harrer  et  al,  2005;  McLaren  et  al,  2004b;  McLaren  et  al, 

2004a;  Jarvis,  Nuzzo-Jones,  &  Heffeman,  2004;  Matsuda,  Cohen,  and  Koedinger,  2005a; 

2005b;  2005c). 

Conclusions 

We  have  made  excellent  progress  toward  creating  and  demonstrating  affordable  behavior 
modeling  through  development  and  evaluation  research  on  the  Cognitive  Tutor  Authoring  Tools. 
The  central  approach  has  been  to  use  human-computer  interaction,  data  mining,  and  machine 
learning  techniques  to  create  software  components,  smart  interfaces  and  underlying  intelligence, 
that  speed  the  process  of  behavior  modeling.  Critical  to  our  success  has  been  creating  a  large 
user  community  that  provides  both  realistic  user  tests  and  vote-with-your-feet  evidence  that 
CTAT  is  effectively  reducing  behavior-modeling  costs.  The  200  users  of  CTAT  represent  the 
largest  number  of  users  of  any  technology  in  the  AHBM  program.  We  have  also  provided 
reasonable  evidence  of  affordability  improvements. 

Significance 

Cognitive  models  are  used  successfully  throughout  the  Armed  Forces  as  well  as  in 
educational  and  training  software.  However,  cognitive  models  have  traditionally  been  very 
expensive  to  build.  This  small  project  has  produced  valuable  knowledge  about  methods  that  can 
be  used  to  make  modeling  more  cost  effective.  This  project  has  also  created  a  specific  set  of 
tools  (CTAT)  that  embodies  some  of  this  knowledge,  and  that  set  of  tools  has  led  to  the  cost- 
effective  creation  of  many  new  cognitive  models.  These  models  in  turn  lead  to  more  knowledge 
as  exhibited  by  the  many  peer-reviewed  papers  produced  under  this  grant  or  related  to  this  grant. 
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International  Conference  on  Advanced  Learning  Technologies  (ICALT  2006). 

Harrer,  A.,  McLaren,  B.  M.,  Walker,  E.,  Bollen,  L.,  and  Sewall,  J.  (2005)  Collaboration  and  Cognitive 
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Patents 

WPI’s  Heffeman  and  CMU’s  Koedinger  filed  for  a  U.S.  Pat.  No.  60/699,624  related  to  the 
WP1  version  of  pseudo-tutor  construction. 

Awards: 

This  grant  funding  helped  lead  to  many  other  successful  projects  that  used  the  results  of  this 
ONR  grant. 

•  Heffeman  was  awarded  an  NSF  CAREER  award  half  way  through  this  grant1. 

•  Koedinger  was  awarded  a  $25  million  NSF  Science  of  Learning  Center  grant  that  is  now 
supporting  the  use,  and  further  development,  of  the  authoring  tools  this  grant  produced. 

•  Heffeman  and  Koedinger  were  awarded  $1 .4  millions  dollars  to  use  the  techniques 
developed  during  the  grant  to  cost-effectively  create  a  complex  new  cognitive  model,  and 
tutoring  systems  used  by  thousands  of  students  inside  of  2  year.  2 3 

•  Heffeman  and  colleagues  at  WPI  were  awarded  a  grant  to  further  the  learning  sciences 

•  Heffeman  and  colleagues  at  Sonalysts  were  funded  to  demonstrate  application  of  these 
techniques  to  help  Fort  Benning  enable  “Warrior  Tutoring”  inside  of  their  simulations  at 
Fort  Benning. 

•  Koedinger  received  a  2-year  grant  from  the  Grable  foundation  to  support  the  further 
development  of  the  CTAT  tool  suite.  Grant  title:  Authoring  Cognitive  Tutors,  Number: 
11655-1-1030164,  Start:  4/12/04,  End:  5/31/06 


1  “Learning  about  Learning”  NSF  CARREER  award  to  Heffeman.  $600,000. 

2  “Using  Web-Based  Cognitive  Assessment  Systems  for  Predicting  Student  Performance  on  State  Exams”  US  Dept 
of  Education:  Institute  of  Education  Sciences  .  $1.4  million. 

3  Title:  “Fellowships  in  CS  to  support  the  learning  sciences  and  security”.  US  Dept  of  Ed:  Graduate  Assistance  in 
Areas  of  National  Need  (GAANN).  $804,940.  PI:  Ward,  Heffeman,  Agu  and  Heineman 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of 
information,  including  suggestions  for  reducing  the  burden,  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188), 
1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any 
penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-MM-YYYY) 
1-02-2006 


4.  TITLE  AND  SUBTITLE 


2.  REPORT  TYPE 


3.  DATES  COVERED  I From  -  To) 

Dec.  1st,  2002-Nov.  30th,  20*05 


Affordable  Cognitive  Modeling  Authoring  Tools  Using  HCI 
Methods:  Worcester  Polytechnic  Institute  Portion 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

N00014-03-1-0221 


6.  AUTHOR(S) 

Heffernan,  Neil,  T. 

Koedinger,  Kenneth,  R. 
Aleven,  Vincent 


5c.  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


51.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Carnegie  Mellon  University 

5000  Forbes  Avenue,  Pittsburgh,  PA  15213 
In  cooperation  with  Worcester  Polytechnic  Institute 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Program  Officer:  Harold  Hawkins 

Office  of  Naval  Research 
800  North  Quincey  Street 
Arlington,  VA  22217-5660 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Unlimited  Distribution 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 

ONR 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


14.  ABSTRACT 

This  is  a  final  report  for  grant  N000140310221  in  which  we  report  on  our  work 


15.  SUBJECT  TERMS 


Cognitive  Modeling,  behavioral  modeling,  cost-effective 


16.  SECURITY  CLASSIFICATION  OF:  7 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

17.  LIMITATION  OF 
ABSTRACT 


18.  NUMBER  19a.  NAME  OF  RESPONSIBLE  PERSON 

Kenneth  R.  Koedinger 

PAGES  - — - - - — - 

,  19b.  TELEPHONE  NUMBER  (Include  area  code) 


412-268-7667 


Standard  Form  298  (Rev.  8/98) 

Prescribed  by  ANSI  Std.  Z3S.18 


